A Statistical-Modelling Approach to Neural Networks

class: title-slide, left, bottom

# A Statistical-Modelling Approach to Neural Networks
----
## **Andrew McInerney**, **Kevin Burke**
### University of Limerick
#### Pukyong National University, 27 March 2024

---
# Background

--

<img src="data:image/png;base64,#img/crt-logo.jpg" width="60%" style="display: block; margin: auto;" />

--

*  Research: Neural networks from a statistical-modelling perspective

--

<img src="data:image/png;base64,#img/packages.png" width="70%" style="display: block; margin: auto;" />

---

# Agenda

--

- Introduction

--

- Model Selection

--

- Penalised Selection
  
--

- Stepwise Selection

--
 
- Model Interpretation

---
class: inverse1 middle center subsection
# Introduction

---

# Background

--

Neural networks originated from attempts to model the human brain.

<br>

--

Early influential papers:

--

-  McCulloch and Pitts (1943)

--

-  Rosenblatt (1958)

--

-  Rumelhart, Hinton and Williams (1986)

---

# Background

Interest within the statistics community in the late 1980s and early 1990s.

--

Comprehensive reviews provided by White (1989), Ripley (1993), Cheng and Titterington (1994).

--

However, majority of research took place outside the field of statistics
(Breiman, 2001; Hooker and Mentch, 2021).

---

# Background

Renewed interest in merging statistical models and neural networks.

--

From a statistical viewpoint:

--

-  Distributional regression (Rugamer et al., 2020, 2021).

--

-  Mixed modelling (Tran et al., 2020).

--

From a machine-learning viewpoint:

--

- Neural Additive Models (Agarwal et al., 2020)

---

# Feedforward Neural Networks

--

.left-column[
<img src="data:image/png;base64,#img/fnn-1.jpg" width="100%" style="display: block; margin: auto;" />
]

<br>

--

.right-column[ 
<img src="data:image/png;base64,#img/nneq1.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---
count: false
# Feedforward Neural Networks

.left-column[
<img src="data:image/png;base64,#img/fnn-1.jpg" width="100%" style="display: block; margin: auto;" />
]

<br>

.right-column[ 
<img src="data:image/png;base64,#img/nneq2.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---
count: false
# Feedforward Neural Networks

.left-column[
<img src="data:image/png;base64,#img/fnn-1.jpg" width="100%" style="display: block; margin: auto;" />
]

<br>

.right-column[ 
<img src="data:image/png;base64,#img/nneq3.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---
count: false
# Feedforward Neural Networks

.left-column[
<img src="data:image/png;base64,#img/fnn-1.jpg" width="100%" style="display: block; margin: auto;" />
]

<br>

.right-column[ 
<img src="data:image/png;base64,#img/nneq4.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---
count: false
# Feedforward Neural Networks

.left-column[
<img src="data:image/png;base64,#img/fnn-1.jpg" width="100%" style="display: block; margin: auto;" />
]

<br>

.right-column[ 
<img src="data:image/png;base64,#img/nneq5.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---

# Data Application

--

### Insurance Data (Kaggle)

--

1,338 beneficiaries enrolled in an insurance plan

--

Response:  
`charges`

--

6 Explanatory Variables:

`age,` `sex,` `bmi,` `children,` `smoker,` `region`

---

# R Implementation

--

Many packages available to fit neural networks in R.

<br>
--

Some popular packages are:

--

-  `nnet`

--

-  `neuralnet`

--

-  `keras`

--

-  `torch`

---

# R Implementation: nnet

--

```r
library(nnet)
nn <- nnet(charges ~ ., data = insurance, size = 2, maxit = 2000,
           linout = TRUE)
summary(nn)
```

--

```{.bg-primary}
## a 8-2-1 network with 21 weights
##  b->h1 i1->h1 i2->h1 i3->h1 i4->h1 i5->h1 i6->h1 i7->h1 i8->h1 
##   1.39  -0.43   0.08   0.03  -0.08  -3.16   0.07   0.11   0.15 
##  b->h2 i1->h2 i2->h2 i3->h2 i4->h2 i5->h2 i6->h2 i7->h2 i8->h2 
##   6.31   0.04   0.13   2.19  -0.11  -6.19   0.15   0.12   0.14 
##  b->o h1->o h2->o 
##  1.08 -4.82  2.45 
## [...]
```

---
# Motivation

--

<img src="data:image/png;base64,#img/insurance_mse_gam-1.png" width="75%" style="display: block; margin: auto;" />

---

# Statistical Perspective

--

$$
y_i = \text{NN}(x_i) + \varepsilon_i, 
$$

--

where

$$
\varepsilon_i \sim N(0, \sigma^2)
$$

--

$$
\ell(\theta, \sigma^2)= -\frac{n}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\text{NN}(x_i))^2
$$

---
class: inverse1 middle center subsection
# Model Selection

---
class: inverse1 middle center subsection
## Penalised Selection

---
# Smooth Information Criterion

--

<img src="data:image/png;base64,#img/sic-publication.png" width="100%" style="display: block; margin: auto;" />

---
# Smooth Information Criterion

$$
 \text{BIC} = -2\ell(\theta) + \log(n) \left[ \sum_{j=1}^p |\beta_j|^0 + 1 \right]
$$

--

where

`\begin{equation*}
  \ell(\theta)= -\frac{n}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-x_i^T\beta)^2
\end{equation*}`

---
# Smooth Information Criterion

$$
 \text{BIC} = -2\ell(\theta) + \log(n) \left[ \sum_{j=1}^p |\beta_j|^0 + 1 \right]
$$

--

Introduce "smooth BIC":

--

$$
 \text{SBIC} = -2\ell(\theta) + \log(n) \left[ \sum_{j=1}^p \frac{{\beta_j^2}}{\beta_j^2 + \epsilon^2} + 1 \right]
$$

---
# Extending to Neural Networks

`$$\mathbb{E}(y) = \text{NN}(X, \theta)$$`

--

where

`$$\text{NN}(X, \theta) = \phi_o \left[ \gamma_0+\sum_{k=1}^q \gamma_k \phi_h \left( \sum_{j=0}^p \omega_{jk}x_{j}\right) \right]$$`

---
# Extending to Neural Networks

<p style="font-size: 0.85em">
$$
 \text{SBIC} = -2\ell(\theta) + \log(n) \left[ \sum_{jk} \frac{\omega_{jk}^2}{\omega_{jk}^2 + \epsilon^2} + \sum_{k} \frac{\gamma_k^2}{\gamma_k^2 + \epsilon^2} + q + 1 \right]
$$
</p>

--
where

<p style="font-size: 1em">
$$
  \ell(\theta) = -\frac{n}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\text{NN}(x_i))^2
$$
</p>

---
# Simulation Setup

<img src="data:image/png;base64,#img/sim-setup.png" width="50%" style="display: block; margin: auto;" />

---
# Results

<img src="data:image/png;base64,#img/sim-single-results.png" width="90%" style="display: block; margin: auto;" />

---
# Extending to Group Sparsity

--

.pull-left[
<img src="data:image/png;base64,#img/input-group.png" width="100%" style="display: block; margin: auto;" />
]

--
Single penalty:

`\begin{equation*}
 \frac{\omega_{jk}^2}{\omega_{jk}^2 + \epsilon^2}
\end{equation*}`

--

Group penalty:

$$
 \text{card}(\omega_j) \times \frac{||\omega_j||_2^2}{||\omega_j||_2^2 + \epsilon^2} 
$$

---
class: inputgroup-slide
# Group Sparsity

## Input-neuron penalization

<p style="font-size: 0.78em">
$$
 \text{IN-SBIC} = -2\ell(\theta) + \log(n) \left[ q \times \sum_{j}\frac{||\omega_j||_2^2}{||\omega_j||_2^2 + \epsilon^2} + \sum_{k} \frac{\gamma_k^2}{\gamma_k^2 + \epsilon^2} + q + 1 \right]
$$
</p>

where `\(\omega_{j} = (\omega_{j1},\omega_{j2},\dotsc,\omega_{jq})^T\)`

---
class: hiddengroup-slide
# Group Sparsity

## Hidden-neuron penalization

<p style="font-size: 0.78em">
$$
 \text{HN-SBIC} = -2\ell(\theta) + \log(n) \left[ (p + 1) \times \sum_{k}\frac{||\theta^{(k)}||_2^2}{||\theta^{(k)}||_2^2 + \epsilon^2} + q + 1 \right]
$$
</p>

where `\(\theta^{(k)} = (\omega_{1k},\omega_{2k},\dotsc,\omega_{pk}, \gamma_k)^T\)`

---
# Simulation Setup

<img src="data:image/png;base64,#img/sim-setup.png" width="50%" style="display: block; margin: auto;" />

---
# Results (IN-SBIC)

<img src="data:image/png;base64,#img/sim-input-results.png" width="90%" style="display: block; margin: auto;" />

---
# Data Application - Results

<img src="data:image/png;base64,#img/insurance.png" width="100%" style="display: block; margin: auto;" />

---
class: inverse1 middle center subsection
## Stepwise Selection

---
class: selectnn-slide
# Model Selection

<img src="data:image/png;base64,#img/modelsel.png" width="90%" style="display: block; margin: auto;" />

A Statistically-Based Approach to Feedforward Neural Network Model Selection (arXiv:2207.04248)

---
class: selectnn-slide
# Insurance: Model Selection

```r
library(selectnn)
nn <- selectnn(charges ~ ., data = insurance, Q = 8,
               n_init = 5)
summary(nn)
```

--

```{.bg-primary}
## [...]
## Number of input nodes: 4 
## Number of hidden nodes: 2 
## 
## Value: 1218.738 
##         Covariate Selected Delta.BIC
##        smoker.yes      Yes  2474.478
##               bmi      Yes   919.500
##               age      Yes   689.396
##          children      Yes    13.702
## [...]
```

---
class: inverse1 middle center subsection
# Model Interpretation

---
# Proposed Solution: interpretnn

--

.left-column[ 
<br>
<img src="data:image/png;base64,#img/interpretnn.png" width="80%" style="display: block; margin: auto;" />
]

--

.right-column[
<br>
<br>
.small[

```r
# install.packages("devtools")
library(devtools)
install_github("andrew-mcinerney/interpretnn")
```
]
]

---

# Significance Testing

--

.pull-left[
<img src="data:image/png;base64,#img/fnn-1.jpg" width="98.9%" style="display: block; margin: auto;" />
]

---
count: false
# Significance Testing

.pull-left[
<img src="data:image/png;base64,#img/fnn2-1.jpg" width="100%" style="display: block; margin: auto;" />
]

--

.pull-right[
  Wald test:
  
  {{content}}
  
  ]

--

$$
`\begin{equation}
 \omega_j = (\omega_{j1},\omega_{j2},\dotsc,\omega_{jq})^T
\end{equation}`
$$
{{content}}

--

$$
`\begin{equation}
 H_0: \omega_j = 0
\end{equation}`
$$
{{content}}

--

$$
`\begin{equation}
 (\hat{\omega}_{j} - \omega_j)^T\Sigma_{\hat{\omega}_{j}}^{-1}(\hat{\omega}_{j} - \omega_j) \overset{\mathcal{D}}{\longrightarrow} \chi^2_q
\end{equation}`
$$
{{content}}

---

# Insurance: Model Summary

```r
intnn <- interpretnn(nn)  
summary(intnn)
```

--

```{.bg-primary}
## Coefficients:
##                                Weights |      X^2    Pr(> X^2)
##               age     (-0.43***, 0.04) |  41.4363 1.01e-09 ***
##          sex.male        (0.08*, 0.13) |   5.5055 6.38e-02 .  
##               bmi      (0.03, 2.19***) | 105.6106 0.00e+00 ***
##          children   (-0.08***, -0.11.) |  19.0146 7.43e-05 ***
##        smoker.yes (-3.16***, -6.19***) | 250.6393 0.00e+00 ***
##  region.northwest        (0.07., 0.15) |   2.8437 2.41e-01    
##  region.southeast        (0.11*, 0.12) |   6.2560 4.38e-02 *  
##  region.southwest       (0.15**, 0.14) |  10.8218 4.47e-03 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
```

---

# Insurance: Model Summary

```r
plotnn(intnn)
```

--

<img src="data:image/png;base64,#img/plotnn-1.png" width="90%" style="display: block; margin: auto;" />

---

# Covariate-Effect Plots

--

$$
`\begin{equation}
 \widehat{\overline{\text{NN}}}_j(x) = \frac{1}{n}\sum_{i=1}^n \text{NN}(x_{i,1}, \ldots,x_{i,j-1},x, x_{i,j+1}, \ldots)
\end{equation}`
$$

--
Covariate-effect plots of the following form:

--

$$
`\begin{equation}
 \hat{\beta}_j(x,d) =\widehat{\overline{\text{NN}}}_j(x + d) -\widehat{\overline{\text{NN}}}_j(x)
\end{equation}`
$$

--

Usually set `\(d = \text{SD}(x_j)\)`

---

# Insurance: Covariate Effects

```r
plot(intnn, conf_int = TRUE, which = c(1, 4))
```

--

.pull-left[ 
<img src="data:image/png;base64,#img/PCE_age-1.png" width="90%" style="display: block; margin: auto;" />
]

--

.pull-right[ 
<img src="data:image/png;base64,#img/PCE_children-1.png" width="90%" style="display: block; margin: auto;" />
]

---
# Summary

--

- Treat neural networks as statistical models

--

- Perform penalised and stepwise model selection

--

- Use hypothesis tests and covariate-effect plots for interpretation

---
class: bigger
# References

*  <font size="5">McInerney, A., & Burke, K. (2022). A statistically-based approach to feedforward neural network model selection. <i>arXiv preprint arXiv:2207.04248</i>.  </font>

*  <font size="5">McInerney, A., & Burke, K. (2023). Interpreting feedforward neural networks as statistical models. <i>arXiv preprint arXiv:2311.08139</i>.    </font>

*  <font size="5">McInerney, A., & Burke, K. (2024). Combining a smooth information criterion with neural networks. <i>To appear on arXiv</i>.    </font>

### R Packages

```r
devtools::install_github(c("andrew-mcinerney/selectnn",
                           "andrew-mcinerney/interpretnn"))
```

<svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:1em;width:0.97em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#007DBA;overflow:visible;position:relative;"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> <font size="5.5">andrew-mcinerney</font>   <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#007DBA;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg> <font size="5.5">@amcinerney_</font> <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#007DBA;overflow:visible;position:relative;"><path d="M64 112c-8.8 0-16 7.2-16 16v22.1L220.5 291.7c20.7 17 50.4 17 71.1 0L464 150.1V128c0-8.8-7.2-16-16-16H64zM48 212.2V384c0 8.8 7.2 16 16 16H448c8.8 0 16-7.2 16-16V212.2L322 328.8c-38.4 31.5-93.7 31.5-132 0L48 212.2zM0 128C0 92.7 28.7 64 64 64H448c35.3 0 64 28.7 64 64V384c0 35.3-28.7 64-64 64H64c-35.3 0-64-28.7-64-64V128z"/></svg> <font size="5.5">andrew.mcinerney@ul.ie</font>